Extracting Motion Annotations from MPEG-2 Compressed Video for HDTV Content Management Applications

Abstract Our research concentrates on developing a novel HDTV content management system that enables end users to search, retrieve, and browse archived standard definition (SD) and high definition (HD) television material for program production and content repurposing in a digital television studio. We have developed the first system that automatically analyzes motion occurring in MPEG-2 coded SD and HD videos within the compressed domain itself, and produces descriptors characterizing the global visual motion in videos, for content-based search and retrieval applications. In this paper, we describe our robust and efficient scheme for automatically generating a flow characterization of a video bitstream without decompression, which is a frame-type-independent uniform motion representation amenable for consistent interpretation and computed from the raw motion vectors encoded in the MPEG2 bitstreams. We propose novel techniques to handle all the different prediction schemes that are employed with different frame types and picture structures of MPEG-2 during the motion compensation process to deal with interlaced and progressive modes. Experiments with thousands of frames from SD and HD video streams demonstrate the accuracy of our flow estimation process and the effectiveness of utilizing flow vectors for annotation of perceived global motion in video streams.